Video-Text Pre-training with Learned Regions for Retrieval

Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information. State-of-the-art approaches extract features raw pixels in an end-to-end fashion. However, these methods operate frame-level directly thus overlook spatio-temporal structure of objects video, which yet has a strong synergy with nouns descriptions. In this work, we propose simple effective module for representation learning, namely RegionLearner, can take into account during on pairs. Given our (1) first quantizes continuous clustering patch-features same cluster according to content similarity, then (2) generates learnable masks aggregate fragmentary regions complete semantics, finally (3) models dependencies different semantic regions. contrast using off-the-shelf object detectors, proposed does not require explicit supervision is much more computationally efficient. We pre-train approach public WebVid2M CC3M datasets. Extensive evaluations four downstream retrieval benchmarks clearly demonstrate effectiveness RegionLearner.